The technology described herein relates to data processing systems, and in particular to the operation of data processing systems that display images on a display.
FIG. 1 shows an exemplary data processing system 14 that comprises a host processor comprising a central processing unit (CPU) 7, a graphics processing unit (GPU) 2, a video codec 1, a camera 12, an image signal processor (ISP) 13, a display processor 5, and a memory controller 8. As shown in FIG. 1 these units communicate via an interconnect and have access to off-chip memory 3. In this system the GPU 2, video codec 1, ISP 13 and/or CPU 7 will generate frames (images) to be displayed and the display processor 5 will then provide the frames to a display 4 for display. The display 4 may be local or remote to the remainder of the system, and may have a wired or wireless connection to the display processor 5.
In use of this system, an application such as game executing on the host processor (CPU) will, for example, require the display of frames on the display 4. To do this, the application 10 will submit appropriate commands and data to a driver 11 for the graphics processing unit 2 that is executing on the CPU 7. The driver 11 will then generate appropriate commands and data to cause the graphics processing unit 2 to render appropriate frames for display and to store those frames in appropriate frame buffers, e.g. in the main memory 3. The display processor 5 will then read those frames into a buffer for the display from where they are then read out and displayed on the display panel of the display 4.
The frames for display may also or instead be streamed to the display processor from the processing unit, e.g. the GPU 2 or ISP 13, that is producing the frames, e.g. through a cache, if desired.
An example of a use of a data processing system 14 such as that illustrated in FIG. 1 is to provide a so-called “XR” display, such as an augmented reality (AR) and/or virtual reality (VR), head-mounted display (HMD) system. (In this case, the display 4 will be a head-mounted display of some kind.)
In a head-mounted AR/VR display operation, the appropriate images to be displayed to each eye will, e.g., be rendered by the graphics processing unit 2, in response to appropriate commands and data from the application, such as a game, (e.g. executing on the CPU 7) that requires the augmented reality or virtual reality display. The GPU 2 will, for example, render the images to be displayed at a rate that matches the refresh rate of the display, such as 30 frames per second.
In such arrangements, the system will also operate to track the movement of the head/gaze of the user (so-called head pose tracking). This head orientation (pose) data is then used to determine how the images should actually be displayed to the user for their current head position (view direction), and the images (frames) are rendered accordingly (for example by setting the camera (view point) orientation based on the head orientation data), so that an appropriate image based on the user's current direction of view can be displayed.
While it would be possible simply to determine the head orientation (pose) at the start of rendering a frame to be displayed in an AR/VR system, because of latencies in the rendering process, it can be the case that the user's head orientation (pose) has changed between the head orientation (pose) at the beginning of the rendering of the frame and the time when the frame is actually displayed (scanned out to the display panel).
To allow for this, frames to be displayed are rendered based on the head orientation data sensed at the beginning of the rendering of the frames, but then before the frames are actually displayed, the frames are re-projected based on an expected head orientation for the time that the frames will actually be displayed to the user. The expected head orientation may be, e.g., based on predicted head movements of the user between the time the frame is rendered and when it will actually be displayed. The re-projected version of the frame is then displayed. This allows the image displayed on the display to more closely match the user's (predicted) current head orientation.
To do this processing, the initial “application” frames are rendered into appropriate buffers in the memory but there is then a second rendering process that takes the initial, application frames in memory and uses the predicted head orientation (pose) information to render “re-projected” versions of the initially rendered frames that take account of the expected head orientation to provide the frames that will be displayed to the user. This typically involves some form of transformation on the frames, based on the predicted head orientation (pose). The so-transformed (re-projected) rendered frames that are actually to be displayed may then be written into a further buffer or buffers in memory from where they are then read out for display by the display processor, and/or streamed to the display processor directly for display.
The transformation (re-projection) of the initial frames based on the predicted head orientation (pose) may be carried out, e.g., by the graphics processing unit 2, and/or by the display processor 5 (where the display processor has appropriate transformation functionality), e.g. under appropriate control from the CPU 7. Thus, for this processing, the graphics processing unit 2 will render the “application” frames as required and instructed by the application, and then the graphics processing unit or the display processor, for example, will re-project those rendered frames appropriately based on predicted head orientation data for processing by the display processor 5 for display.
The Applicants believe that there is scope for improved arrangements for displaying images in head-mounted display systems, such as for augmented and/or virtual reality applications.
Like reference numerals are used for like components throughout the drawings, where appropriate.